Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add net/http/pprof profiling endpoint to http port #913

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Add net/http/pprof profiling endpoint to http port #913

wants to merge 6 commits into from

Conversation

timvaillancourt
Copy link
Member

@timvaillancourt timvaillancourt commented Jul 6, 2023 •
edited
Loading

This PR adds the ability to gather pprof profiles from Nebula over HTTP in order to understand/improve performance and understand the high CPU usage we see in Slack's production Vitess platform

This change requires making the http server opened by stats.go to become more generic than "just stats", so some logic has been shuffled into http.go and a new, generic config variable http.listen has been added to define this port

For backwards compatibility, http.listen will fallback to using the value for stats.listen if http.listen is undef. I suggest the stats.listen var (and the fallback) is removed in the future because the http port will not be just stats after this PR

@timvaillancourt timvaillancourt marked this pull request as draft July 6, 2023 17:44
@timvaillancourt
Add http pprof profiling endpoint
dd7f12f 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt
remove dupe stats.path check
90aa8ef 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt timvaillancourt marked this pull request as ready for review July 6, 2023 23:41
@timvaillancourt
Fix typo in log
70779b2 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt
Cleanup code
af95867 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt
Cleanup code comment
c14ad0f 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt timvaillancourt changed the title Add http pprof profiling endpoint Add net/http/pprof profiling endpoint to http port Jul 7, 2023
@wadey
Copy link
Member

wadey commented Jul 7, 2023

Thanks! We will consider this, but I'm not sure if you are aware there is already a method to get pprof traces using the SSH debug interface.

Once you connect you can run help to see the methods, but the ones you might care about are start-cpu-profile, stop-cpu-profile.

Defined here in the code: https://github.com/slackhq/nebula/blob/v1.7.2/ssh.go#L228

@wadey wadey added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jul 7, 2023
@timvaillancourt
improve comments
0af7e6a 
Signed-off-by: Tim Vaillancourt <[email protected]>
@timvaillancourt timvaillancourt changed the title Add net/http/pprof profiling endpoint to http port Add net/http/pprof profiling endpoint to http port Jul 7, 2023
@timvaillancourt
Copy link
Member Author

Thanks! We will consider this, but I'm not sure if you are aware there is already a method to get pprof traces using the SSH debug interface.

Once you connect you can run help to see the methods, but the ones you might care about are start-cpu-profile, stop-cpu-profile.

Defined here in the code: https://github.com/slackhq/nebula/blob/v1.7.2/ssh.go#L228

Thanks @wadey! I didn't notice that when I started this branch, but saw mentions of it in issues later 👍

It seems we don't enable the sshd server in Prod (although we could), so this isn't currently an option. That said, I'd much prefer to use the standard net/http/pprof approach because it would not require adding sshd to all our hosts, it is easier to plug-in to our automation that already supports fetching profiles via the standard http approach, and /debug/pprof provides access to more information than CPU profiles, such as heap and mutex usage, tracing, etc

@nbrownus nbrownus mentioned this pull request Jul 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@timvaillancourt @wadey